Abstract Title Page 

Not included in page count. 


Title: The Development and Testing of a New Measure of Early Childhood Education 
Organizational Conditions 

Authors and Affiliations: 

Stacy B. Ehrlich, University of Chicago Consortium on Chicago School Research 
Debra M. Pacchiano, The Ounce of Prevention Fund 

Stuart Luppescu, University of Chicago Consortium on Chicago School Research 
Amanda Stein, The Ounce of Prevention Fund 


SREE Spring 2016 Conference Abstract Template 



Abstract Body 


Background/Context: 

The early education field, like K-12, faces substantial challenges improving the quality of 
teaching and learning and, thus, student outcomes. Despite decades of evidence that high-quality 
early education can positively impact the learning trajectories of low-income children, the 
majority of programs nationwide fail to significantly advance children’s achievement (Belfield, 
Nores, Barnett, & Schweinhart, 2006). With the intention of supporting improvement efforts, the 
early education field has reliable and valid tools to measure classroom structural quality, 
classroom interactions and instruction, interactions with families, administrative practices, and 
program culture and climate (e.g., Bryant, 2010). 1 However, these existing tools measure 
discrete constructs of quality, rather than the complex, integrated organizational structures that 
work in tandem to either impede or support the effective practice of educators (Zaslow, Tout, & 
Martinez-Beck, 2010). Educational research has greatly advanced our understanding of this 
complexity and the dynamics of improvement beyond the individual elements of quality. 

Despite these advances, the field lacks a rigorously developed tool to measure organizational 
conditions in early education settings, the results of which can then focus leaders’ attention on 
these organizational dynamics and provide them with accessible and actionable information to 
improve these conditions. This proposed paper describes the development and pilot testing of 
this missing tool — what we refer in this proposal as “ Early Childhood Surveys .” 

Prior research on elementary schools led to the identification of five school-level organizational 
constructs empirically linked to school improvement, articulated in the “five essentials 
framework” (Bryk, Sebring, Allensworth, Luppescu, & Easton, 2010). These five constructs are: 
effective leadership, collaborative teachers, family ties, supportive environment, and ambitious 
instruction. These researchers developed teacher and student surveys to measure these five 
essential supports and found that they strongly predicted which schools were most and least 
likely to improve over time: Schools strong in three or more of these supports were 10 times 
more likely than schools weak in most supports to substantially improve student math and 
reading outcomes. Furthermore, a weakness in just one of these supports over time undermined 
other change efforts, reducing the probability of improvement to 10% or less (Bryk et ah). 

Early education research suggests, as in elementary settings, that programs more successfully 
promote children’s learning and development when they have strong organizational practices 
aligned to these five constructs (Burchinal, Vandergrift, Pianta, & Mashburn, 2010; Rohacek, 
Adams, Kisker, Danziger, Derrick-Mills, & Johnson, 2010; West-Olatunji, Behar-Horernstein, & 
Rank, 2008). The authors therefore posit that programs with strong practices aligned to these 
essentials also will exhibit higher-quality classroom practices and better prepare children for 
kindergarten. The current project, therefore, seeks to adapt the existing five essentials teacher 
surveys, and create a new parent survey, for use in early education settings. 

Purpose / Objective / Research Question / Focus of Study: 

The current study sought to scientifically adapt the existing five essentials teacher survey and 
develop a new parent survey, and pilot test them for use in early childhood education settings. 


1 Examples of tools measuring these discrete constructs are: Early Childhood Environment 
Rating Scales (ECERS-R); (Hanns, Clifford, & Cryer, 2005); Classroom Assessment Scoring 
System (CLASS); (Pianta, LaParo, & Hamre, 2007); Parent Education Profile (PEP); (RMC 
Research Corporation, 2006); Program Administration Scale (PAS): (Talan & Bloom, 2004). 
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The adaptation and development of these Early Education Surveys occurred in 2014-15, was 
based strongly on Bryk et al.’s (2010) existing framework, and utilized the existing 5Essentials 
surveys, which are publicly-available through the Consortium on Chicago School Research’s 
website, with author permission. This proposed paper presents the development process of these 
two surveys and pilot test findings. In the future, our work will test whether these surveys 
measure organizational elements of a preschool program that are related to positive classroom 
and child outcomes via a validation study (to be conducted in 2015-16). Ultimately, the purpose 
of the Early Education Surveys is to provide reliable and valid survey data that will guide early 
childhood educators to generate continuous quality improvements in teaching and learning by 
strengthening the organizational conditions in which teachers work every day with children and 
families. 

Setting: 

The Early Education Surveys have been developed for use in, and therefore were piloted in, 
publicly-funded early education programs with at least 3 classrooms serving children ages 3 to 5 
in either school- or community-based settings. Data collection took place in a large, urban city 
in the Midwest. Focus groups and cognitive interviews took place with teachers and parents in 
both school-based and center-based preschool programs across this city. Pilot study data 
collection occurred in a variety of locations. The teacher survey was piloted with preschool 
teachers in a local, urban school district as well as in Head Start programs across the nation. The 
parent survey was piloted in the same local, urban school district as well as center-based Head 
Start programs across this local city. 

Population / Participants / Subjects: 

For the Early Education Teacher Survey, the authors conducted a single pilot study in Spring 
2015. To pilot in school-based programs, the survey was offered to all preschool teachers within 
our local school district; a total of 1,153 teachers responded. Teacher pilot data were also 
collected from a national sample of center-based Head Start teachers participating in the Family 
and Child Experiences Survey (conducted by the Office of Program, Research, and Evaluation 
Office at the Department of Health & Human Services). Eighty-eight percent of those teachers 
sampled (n=363) responded to the survey (see Table 1 for total numbers). 

To develop the Early Education Parent Survey, we embarked on two rounds of interviews with 
parents and staff, pilot data collection, and Rasch analyses. Our first round (Fall 2014) included 
focus groups with early education teachers, early education family support specialists, and 
parents; cognitive interviews with 16 parents; and survey piloting in 4 community-based 
programs, resulting in 198 pilot surveys from English- and Spanish-speaking parents. Our second 
round of development included revisions based on our initial Rasch analyses and additional 
cognitive interviews with parents. The final pilot of the parent survey was conducted in Spring 
2015, with surveys administered on-site in seven school-based preschool programs and nine 
community-based preschool programs. A total of 229 parents, roughly half English-speaking and 
half Spanish-speaking (evenly distributed over school- and community-based sites), completed 
the surveys (see Table 2); 81% were completed online using tablets (the remaining were hard 
copy versions, mostly due to internet connection issues). 

Research Design: 

Generating the Early Education Surveys required an iterative development and testing process; 
we used qualitative and quantitative feedback from one cycle to infonn the next phase of 
development and testing. Our method of survey development includes the creation of measures. 
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with corresponding items that comprise them, which are analyzed using the Rasch model 
(Wright & Masters, 1982). To create the Early Education Surveys, our study required the 
adaptation of existing teacher surveys and the development of a new parent survey. For the 
teacher survey, the authors relied on a bank of existing measures and items publicly available 
from the 5EssentiaIs survey (CCSR, 2014). The authors detennined whether existing teacher 
items fit into one of three categories: (a) items that were appropriate as-is for the early education 
context, (b) items that needed language and terminology revisions for early education, and (c) 
items that are irrelevant to early education and, therefore, should be discarded. Then, we 
established whether there were concepts critical in early education not addressed in the existing 
5Essentials surveys (see Figure 1). For the parent survey, the authors conducted additional 
information collection activities (focus groups, cognitive interviews; see next section) as well as 
an initial round of pilot testing prior to the final pilot of both surveys, conducted in Spring 2015 
(see Figure 2). 

The development and pilot study for the Early Education Surveys included the following design 
methods: (1) Focus groups. To determine appropriateness for the early education context, we 
conducted focus groups with four stakeholder groups: (a) exemplar leaders and supervisors from 
publicly-funded programs in our local city; (b) academic scholars who focus on effective early 
education (c) parents who are involved in their program’s Head Start Policy Council, Parent 
Committee, Local School Council, or local community organizations; and (d) early education 
teachers and family engagement staff. Conducting focus groups allowed us to carefully attend to 
the terminology used in our items to ensure they evoke the intended constructs within early 
education (ensuring content validity). (2) Cognitive interviews. To test our revised or newly 
generated items, we conducted cognitive interviews. Cognitive interviewing is a method of pre- 
testing, involving one-on-one interviews with individuals in the target population. Respondents 
were asked survey questions in a semi-structured fonnat to explore respondents’ thought 
processes and challenges answering the question. This method has been found to be an 
inexpensive and effective way of identifying problems in surveys (Conrad & Blair, 2004; Willis, 
2005). Cognitive interviews ensure that items are easy to understand, the question stems and 
answer choices are unambiguous, and the questions asked are being interpreted as intended. This 
provided a second opportunity to ensure content validity. (3) Pilot testing and Rasch analysis. 
Rasch measurement is a model that creates unidimensional measures, measuring how people feel 
on a single, unitary concept. After each round of pilot data collection, we conducted Rasch 
analyses, attending specifically to measure reliability, indicators of misfitting items, mismatches 
between item difficulty order and theoretical expectations, and alignment between item and 
person difficulties. Results of our analyses led to item and measure revisions. Using Rasch 
modeling, we determined whether the measures included in our pilot study were reliable and 
differentiated among survey-takers. 

Data Collection and Analysis: 

The proposed presentation will describe the process of survey development but will focus largely 
on the data collection, Rasch analysis, and results of the Spring 2015 pilot study. The school- 
based teacher surveys were administered as part of a larger, online-only survey administration 
across a local school district from January through April 2015. An item early in the survey 
identified whether a teacher was a preschool teacher, allowing preschool-specific items to be 
made available to them. The center-based teacher surveys were administered as part of a larger, 
national study on Head Start programs in Spring 2015. Surveys were available online or could be 
requested as a hard copy. All teacher surveys were provided in English only. 
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The school-based and center-based parent surveys were collected by the authors and research 
team members. Teams of data collectors visited sites and recruited parents to take the pilot 
survey between May- June 2015. The online survey was developed using Qualtrics and tablets 
were available to parents for use; hard copy versions were available upon request or used when 
difficulties arose connecting to the internet. Surveys were available in both English and Spanish. 

Findings / Results: 

Rasch analyses were conducted on measures that were pilot tested with teachers and parents. The 
authors revised measures based on the results to improve the performance of measures (e.g., 
removing items that misfit). Tables 3 and 4 provide resulting measure reliabilities. 60% of both 
our teacher and parent measures had a reliability >=.80 (separation >2). Those with reliabilities 
<.80 will either be dropped or will be revised prior to conducting a full validation study. In most 
cases, low reliability seemed to be because the measure did not include items difficult enough to 
capture the full range of respondents. Once final measures are created, we will conduct 
differential item function (DIF) analyses to explore whether there are significant differences 
between responses in school-based or center-based programs, and whether there were differences 
by parents who responded to English and Spanish surveys. 

Conclusions: 

In practice, it is extremely difficult to move the needle on student outcomes at scale. Similarly, 
research on improvement efforts often is not able to identify the conditions that support or hinder 
implementation of these efforts. The development of the Early Education Surveys will provide 
needed contextual information that will inform the work of both practitioners and researchers. 
Our aim is for the surveys to continue to be both informed by practice and research and influence 
practice. Specifically, these surveys will provide leaders and practitioners with actionable 
information to diagnose the strengths and weaknesses of programs so they can build on those 
strengths and focus improvement efforts on strengthening weaknesses. Simultaneously, the 
surveys will provide researchers with a validated instrument for future education research and 
evaluation. For example, the Early Education Surveys will enable deeper investigation into the 
factors that mediate or moderate the effects of organizational strength and change on classroom 
and child outcomes. It will also provide researchers with a new assessment of previously 
unmeasured interdependent constructs in early education. This will enable researchers studying 
early education interventions to evaluate the impact of organizational conditions on 
implementation fidelity and intervention effectiveness. 

However, before these surveys make their way into the field, the authors are taking a rigorous 
research approach. Next steps include a full validation study to measure and test whether the 
Early Education Surveys are related to classroom practices and child outcomes. Efforts will be 
made to understand what high- and low-levels of implementation of these five organizational 
constructs looks like in different settings, further adding knowledge to the early education field. 
As we continue our research, we are engaging continuously and closely with practice leaders — 
including the top administrators of Head Start in our city, the local school district, the state-level 
early education office, and federal research and practice leaders — in efforts to make these tools 
relevant and useful for use in early education settings. 
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Appendix B. Tables and Figures 


Figure 1. Development Process for Early Education Teacher Survey 
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Figure 2. Development Process for Early Education Parent Survey 
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Table 1. Total Samples for Teacher Pilot Study 
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Table 2. Total Samples for Parent Pilot Study 



TOTAL n 

School-based programs 

Center-based (Head 
Start) programs 



Online 

Paper- 

pencil 

Paper- 

Online pencil 

English 

118 

46 

15 

57 

0 

Spanish 

111 

22 

28 

55 

6 

TOTAL 

229 

68 

43 

112 

6 


Table 3. Preliminary reliability coefficients for piloted Early Education 
Teacher Survey 


Measure Reliability Coefficient 


Essential: Ambitious Instruction 

Quality of Student Interaction 

.81 

Early Math 

.78 

Early Language and Literacy 

.83 

Early Cognitive Development 

.81 

Early Social-Emotional Development 

.69 

Essential: Supportive Environment 

Positive Learning Climate 

.73 

Teacher Safety 

.86 

Attendance 

.71 

Child-Child Interactions 

.79 

Essential: Involved Families 

Teacher-Parent Trust 

.86 

School/Center Welcoming Families 

.69 

Teacher Outreach/Collaboration with Parents 

.74 

Essential: Collaborative Teachers 

Teacher-Teacher Trust 

.81 

Socialization of New Teachers 

.62 

School Commitment 

.76 

Reflective Dialogue 

.83 

Quality Professional Development 

.81 

Innovation 

.83 

Data Use 

.80 

Collective Responsibility 

.89 

Teacher Collaboration 

.83 

Essential: Effective Leaders 

Teacher-Principal Trust 

.90 

Program Coherence 

.69 

Instructional Leadership 

.86 

Teacher Influence 

.84 
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Table 4. Preliminary reliability coefficients for piloted Early Education 
Parent Survey 


Measure 

Reliability 

Coefficient 

Essential: Supportive Environment. 

Child-Child Interactions 

.51 

Support for Kindergarten Transition 

.55 

Program Orientation Towards Early Education 

.84 

Essential: Involved Families 


Parent-Teacher Trust 
Family Engagement in Center/School 
Quality of Engagement Opportunities 
Curriculum-related Teacher Community w/ Parents 
General Teacher Communication with Parents 
Teacher Care and Responsiveness Towards Parents 


Essential: Effective Leaders 


Principal/Director-Parent Relations 

.89 


.96 

.68 

.61 

.95 

.90 

.80 
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